Methods to compensate manufacturing variations and design imperfections in a display device

ABSTRACT

A display device includes (a) a non-volatile memory containing corrective data for compensating input image data received; (b) display hardware receiving control and data signals for displaying an image; and (c) an image processing circuit that retrieves the corrective data from the non-volatile memory to generate the data signals for the display hardware, after applying the corrective data to each color component of each pixel in the input image data.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation-in-part application ofcopending U.S. patent application Ser. No. 14/624,270, entitled “Methodsto Compensate Manufacturing Variations and Design Imperfections in aCapsule Camera,” filed Feb. 17, 2015, which is a continuation of U.S.patent application Ser. No. 11/621,486, entitled “Method to CompensateManufacturing Variations and Design Imperfections in a Capsule Camera,”filed on Jan. 9, 2007, now U.S. Pat. No. 9,007,478.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to methods for compensating for variationsin a manufacturing process. In particular, the present invention relatesto applying such methods to optical instruments.

2. Discussion of the Related Art

The capsule camera has proved to be an effective minimally-invasive wayfor examining the gastrointestinal (GI) tract. One example of aswallowable capsule camera is described in the U.S. Pat. No. 5,604,531,issued to the Ministry of Defense, State of Israel. A number of patentsassigned to Given Imaging describe in greater detail a capsule camerasystem which uses a transmitter to send images captured by the camera toan external receiver. Other capsule camera examples include U.S. Pat.Nos. 6,709,387 and 6,428,469. There are also other patents to theOlympus Corporation regarding capsule cameras. For example, U.S. Pat.No. 4,278,077 discloses a film camera in a capsule designed for thestomach. U.S. Pat. No. 6,939,292 discloses a capsule camera with abuffering memory and a transmitter. U.S. Pat. No. 6,800,060 discloses acapsule camera which stores image data in an atomic resolution storage(ARS) device.

Various advancements in electronics may now be incorporated into thecapsule camera. For example, an LED provides a compact and low-powerlighting source. CMOS image sensors reduce the component count andpower. The further miniaturizing of integrated circuit geometry allowsSOC (system-on-a-chip) techniques to be used, which reduce the size andpower of the capsule camera. However, size reduction pushes the limitsof manufacturing tolerances. For example, manufacturing processes forsemiconductor image sensors inherently include not only lot-to-lot,wafer-to-wafer, and die-to-die variations, but also pixel-to-pixelvariations within a die. Even the analog circuits sensing the chargestored in a cell to provide digital image data vary from location tolocation within a die. In addition, LEDs are known for variations intheir light spectrum and intensity. Also, because the center of a lenspasses light better than at the edge, a higher intensity results in thecenter of the image than at the edges. There are also variations fromlens to lens, even if the lenses are fabricated from the same mold.

Another example results from the fact that illumination sources in acapsule camera illuminate objects only a few centimeters away, theobjects receive non-uniform light intensities. Moreover, the transparentwindow provided in the capsule housing for the LED light to pass throughis not perfectly transparent. Thus, when the light sources in thecapsule (e.g., LEDs) are turned on, the transparent window reflects someof the light within the camera's field of view back to the light sourcesand the image sensor.

As the capsule camera is intended to allow a physician to examine theinside of a patient's GI tract, its accuracy and image quality are ofparamount concern.

SUMMARY

According to one embodiment of the present invention, a display deviceincludes (a) a non-volatile memory containing corrective data forcompensating input image data received; (b) display hardware receivingcontrol and data signals for displaying an image; and (c) an imageprocessing circuit that retrieves the corrective data from thenon-volatile memory to generate the control and data signals for thedisplay hardware, after applying the corrective data to each colorcomponent of each pixel in the input image data.

According to one embodiment of the present invention, the correctivedata in the non-volatile memory is generated from a calibration stepduring manufacturing by a diagnostic system detecting an intensity ofeach color component in each pixel of the display hardware. Thecalibration provides corrective data which enables compensation to theinput image data over a range of intensities for each color component ofeach pixel displayed by the display hardware.

According to one embodiment of the present invention, the display devicefurther includes a thermometer for measuring an operating temperature inthe display hardware. In that embodiment, the corrective data enablescompensation of the input image data based in part on the measuredtemperature.

According to the present invention, the corrective data may be stored inany suitable format. For example, in one embodiment, the corrective datais stored in the non-volatile memory in a table format. The correctivedata in that case may correspond to the corrected values for replacingcorresponding values in the received input image data. Alternatively,the corrective data may correspond to compensation values to be appliedto corresponding values in the received input image data. As anotherexample, the corrective data may correspond to corrective values forpredetermined values of input image data. In that case, correctivevalues for the received input image data are interpolated orextrapolated from the corrective values for the predetermined values. Asone further example, the corrective data may correspond to correctivevalues for predetermined values of input image data and values ofparameters of a model. In that example, corrective values for thereceived input image data are generated from the values of theparameters based on the corrective values for the predetermined values.

According to one embodiment of the present invention, the correctivedata may correspond entirely to values of parameters of a model fromwhich corrective values are generated based on the correspondingreceived input image data. The model may be any suitable function, suchas a linear function or polynomial function.

According to one embodiment of the present invention, the storedcorrective data includes only corrective values that are each applicableto one of a number of specific groups of neighboring pixels in the imagehardware. In that case, each corrective value is applied to each pixelof the corresponding group of neighboring pixels.

The present invention is applicable to correcting manufacturingvariations in the digital domain, as well as the analog domain.

According to one embodiment of the present invention, a method forcorrecting input image data received into a display device formanufacturing variations includes (a) generating corrective data from acalibration step during manufacturing by a diagnostic system detectingan intensity of each color component of each pixel of display hardwarein the display device; (b) storing the corrective data into anon-volatile memory in the display device; and (c) at a subsequent time,(i) retrieving the corrective data from the non-volatile memory; (ii)applying the corrective data to correct each color component of eachpixel in the received input image data; and (iii) generating the controland data signals for the display hardware, based on the corrected inputimage data.

The present invention finds application not only in capsule cameras. Thepresent invention is applicable to cameras and display devices, ingeneral.

The present invention is better understood upon consideration of thedetailed description and the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of a capsule camera in the GI tract.

FIG. 2 illustrates a method for characterizing pixel-to-pixel variationsand design imperfection in a capsule camera, in accordance with oneembodiment of the present invention.

FIG. 3 illustrates a method for characterizing pixel-to-pixel variationsand design imperfection in a capsule camera under its own illumination,in accordance with one embodiment of the present invention.

FIG. 4 shows a method for using a capsule camera using thecharacterization procedures described above, in accordance with oneembodiment of the present invention.

FIG. 5 illustrates an alternative embodiment of the present inventiondescribed in conjunction with FIG. 4.

FIG. 6 illustrates an alternative embodiment of the present inventiondescribed in conjunction with FIG. 4.

FIG. 7 illustrates the alignment between an image sensor array and thefootprint of an image produced by an optical system, such as the lenssystem of a capsule camera.

FIG. 8A shows a Lambertian curve of a radiation pattern of an LED (i.e.,angle versus illuminance) and FIG. 8B the luminance of LED versus angle.

FIG. 9 shows a grid-distortion plot of a lens.

FIG. 10 shows the structures in display 1100 provided for compensatingpixel values in the digital domain, according to one embodiment of thepresent invention.

FIG. 11 shows the structures in display 1200 provided for compensatingpixel values in the analog domain, according to one embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows an exemplary capsule camera in the GI tract. As shown inFIG. 1, swallowable capsule camera system 01 is seen inside body lumen00, which may be, for example, the colon, the small intestines, theesophagus, or the stomach. Capsule camera system 01 is entirelyautonomous while inside the body, with all of its elements encapsulatedin a capsule housing 10 that provides a moisture barrier, protecting theinternal components from bodily fluids. Capsule housing 10 includes atleast a transparent window, so as to allow light from the light-emittingdiodes (LEDs) of illuminating system 12 to pass through the wall ofcapsule housing 10 to illuminate the walls of lumen, and to allow thescattered light from the lumen 00 walls to be collected and imagedwithin the capsule camera. Capsule housing 10 also protects lumen 00from direct contact with the foreign material inside capsule housing 10.Capsule housing 10 is provided a shape that enables it to be swallowedeasily and later to pass through the GI tract efficiently. Generally,capsule housing 10 is sterile, made of a non-toxic material, and issufficiently smooth to minimize the chance of lodging within the lumen.

As shown in FIG. 1, capsule camera system 01 includes illuminatingsystem 12 and a camera that includes optical system 14 and image sensor16. An image captured by image sensor 16 may be processed by imageprocessor 18, which performs various image processing functions, such asdetermining whether or not the capsule is moving relative to the portionof the GI tract within the optical view of the camera. Image processor18 may be implemented by a digital signal processor (DSP) or a centralprocessing unit (CPU). Image processor 18 may have one or more partialframe buffers. A semiconductor non-volatile archival memory 20 may beprovided to allow the images to be retrieved throughout output port 28at a docking station or workstation outside the body, after the capsuleis recovered. The image in archival memory 20 may be stored in acompressed form, using data compression module 22. (Data compressionmodule 22 may be implemented in hardware, or in software running onimage processor 18). Capsule camera system 01 is powered by batterypower supply 24. Capsule camera system 01 may be propelled through theGI tract by peristalsis.

Illuminating system 12 may be implemented by LEDs. In FIG. 1, the LEDsare located adjacent the camera's aperture, although otherconfigurations are possible. The light source may also be provided, forexample, behind the aperture. Other light sources, such as laser diodes,may also be used. Alternatively, white light sources or a combination oftwo or more narrow-wavelength-band sources may also be used. In oneembodiment, white LEDs may be formed by various components, such as ablue LED or a violet LED, along with phosphorescent materials that areexcited by the LEDs' light to emit light at longer wavelengths. Theportion of capsule housing 10 that allows light to pass through may bemade from bio-compatible glass or polymer.

Optical system 14, which may include multiple refractive, diffractive,or reflective lens elements, provides an image of lumen 00 within itsfield of view on image sensor 16. Image sensor 16 may includecharged-coupled devices (CCD) or complementary metal-oxide-semiconductor(CMOS) type sensor devices that convert the received light intensitiesinto corresponding electrical signals. Image sensor 16 may have amonochromatic response or include a color filter array to allow a colorimage to be captured (e.g. using the RGB or CYM color spacerepresentations). The analog signals from image sensor 16 are preferablyconverted into digital form to allow use of digital processingtechniques. Such a conversion may be accomplished using ananalog-to-digital (A/D) converter inside image sensor 16 (as in thecurrent case), or at another location inside capsule housing 10. The A/Dunit may be provided between image sensor 16 and the rest of the system.LEDs in illuminating system 12 are synchronized with the operations ofimage sensor 16. One function of control module 26 (not shown) is tocontrol the LEDs during image capture operation.

Image processor 18 may select an image to retain when the image showsenough motion relative to a previous image in order to conserve thelimited storage space available. The captured images may be stored inon-board archival memory system 20. Output port 28 shown in FIG. 1 isnot operational in vivo, but uploads data to a workstation after thecapsule camera is recovered from a journey through the body.

When the LEDs are turned on, their light passes through the transparentwindow in capsule housing 10 to illuminate objects in the GI tract. Thereflected light from these objects passes through the transparent windowand reaches lens 14, which focuses the image on imaging sensor 16. Someof the light emitted from the LEDs is reflected back from from thesurface of the transparent window, as shown in long-short dash line 103,through lens 14 onto image sensor 16. Because every pixel in a cellarray in image sensor 16 is in some sense unique, image sensor 16 may becharacterized under a uniform lighting condition providedexternally—with the LEDs in capsule camera system 01 turned off—todetermine pixel-to-pixel variations in image sensor 16. In fact, thedifferent readouts of these pixels represent both the pixels' variationsand the lens' contribution to non-uniform pixel values.

FIG. 2 illustrates a method for characterizing pixel-to-pixel variationsand design imperfections in a capsule camera, in accordance with oneembodiment of the present invention. As shown in FIG. 2, a capsulecamera including at least a lens system, an image sensor and atransmitter or an archival memory is prepared (step 201). Next, at step202, objects illuminated by light of a known color, intensity, contrastor pattern is placed in the field of view of the lens system andexposures are made. These images are then used to characterize orcalibrate the capsule camera system. The images may be stored in thearchival memory or transmitted. At a subsequent time (step 203), theseimages are used to calibrate the images captured from a patient forclinical diagnosis purposes.

One method for compensating variations in pixel responsivity and thefield dependence on the camera's response (i.e., the non-uniformity insensor illumination, even in the presence of a uniform objectillumination) is to present the camera with a field of uniform color andbrightness during calibration. A test image is then taken under thisillumination condition and used to obtain an average value AR for thered component of all pixels. If a particular pixel's red componentsignal is PR_(i), the red component of this pixel may be compensatedduring normal operation by multiplying the red component of the measuredpixel value by a factor A_(R)/PR_(i). Other color components may becompensated in a like manner. (This method may also be applied likewiseto color components in other color space domains).

Characterization of the capsule camera system under its own illumination(e.g., illumination system 12) may be carried out. FIG. 3 illustrates amethod for characterizing pixel-to-pixel variations and designimperfection in a capsule camera under its own illumination, inaccordance with one embodiment of the present invention. The “whitebalance” of the combined illumination and imaging system can becorrected. As shown in FIG. 3, a capsule camera including at least alens system, an image sensor, an illumination system within the capsulecamera's housing, and a transmitter or an archival memory is prepared(step 301). Next, at step 302, objects of known color, contrast, andpattern are illuminated by the illumination system in the field of viewof the lens system and exposures are made. These images are then used tocharacterize or calibrate the capsule camera system. The images may bestored in the archival memory or transmitted. At a subsequent time (step303), these images are used to calibrate the images captured from apatient for clinical diagnosis purposes.

During calibration, images may be taken of various colored fields. Theaverage red, green, and blue pixel responses for each test field can becompared to an ideal responses and correction factors determined. Forexample, if a measured average value for the red component is A_(R) andthe expected average pixel value is A_(R) ^(E) then the factor A_(R)^(E)/A_(R) may be used to correct color fidelity—by multiplying eachpixel in a measured image by this factor. Similar factors may be derivedfor the green and blue components. Different test color fields may yielddifferent calibration factors for the three primary colors.Alternatively a white field may be used to calibrate all colors.Multiple images of the same test pattern may be taken and the resultsaveraged so that time-dependent noise is reduced in the measured pixelvalues. A weighted average of calibration factors over the testconditions can be determined and stored for later image correction. Thecompensation factors can be stored in capsule the camera system 01, inthe data recorder, or in the workstation software. The compensationfactors may each be defined using an equation, a curve, a piecewisecurve or a lookup table. The number of parameters per pixel may also bemore than one.

Within image sensor 16, each pixel value is also affected by effects notrelating to light input. Such effects may be seen in an output value ofa “dark” pixel. A dark pixel may be a pixel not within the field of viewof the optical system 14 and which is typically covered by an opaquematerial. An output value of such a pixel is not attributed to lightinput, but effects such as thermally-induced charge leakage. Anoperational pixel may be compensated by adding as an offset thethermally-induced leakage current measured at a dark pixel (or theaverage thermally-induced leakage current of a group of dark pixels).The thermally-induced leakage current is a function of both temperatureand time. The temperature of a capsule camera in operation is expectedto be relatively fixed at the human body temperature, plus a few degreesdetermined by both the power consumption of the capsule camera and thethermal conductance. As a first-order estimate, the thermally-inducedleakage current is proportional to the time between a pre-charge of thepixel and the readout.

The pixel-to-pixel variations or non-uniform pixel responsivity,including the effects of lens system 14, may be compensated according tothe following equations (1) for the R, G and B color components,respectively:

R=F _(R) ×P _(R) +C _(lR) ×t _(l) +C _(R)

G=F _(G) ×P _(G) +C _(lG) ×t _(l) +C _(G)   (1)

B=F _(B) ×P _(B) +C _(lB) ×t _(l) +C _(B)

where, P_(R), P_(G), and P_(B), are the measured component values forthe three color components read from a pixel in image sensor 16, F_(R),F_(G), and F_(B) are multiplication factors provided for correctingpixel-to-pixel variations and for adjusting the average pixel value, andC_(R), C_(G) and C_(B) are offsets—which may be positive or negative—foreach pixel. In this model, the pixel response is assumed to be linearlydependent on the received radiant energy absorbed by the pixel. Theslope of the linear dependency is the “responsivity.” At a dark pixel(i.e., at zero input radiant energy), the zero-light response (“darkcurrent”) represent an offset. It is convenient to use multiplicativefactors for compensation because of its simplicity. However, if thepixel characteristics are non-linear with respect to the componentvalues, a more sophisticated compensation function may be required. Inthe general case, the term F_(k)×P_(k), k=R, G, B may be replaced byf_(k)(P_(k))). Such a function may be, for example, a polynomial, or afunction that includes logarithmic or exponential terms. Of course,f_(k)(P_(k)) may also be a function of other parameters—such astemperature, as discussed below. R, G, and B are the primary colors.P_(R), P_(G) and P_(B) are non-negative values (each value is clamped tozero, when its calculated value is negative, or clamped to 2^(N)−1 for Nbit resolution, when the calculated value exceeds 2^(N)−1).

Parameters C_(lR), C_(lG) and C_(lB) are provided to compensate for thethermally-induced leakage current. The contribution of the leakagecurrent to each color component is proportional to time t_(l), which isthe time duration between cell pre-charge and cell readout. In order tosolve for F_(R), C_(lR), and C_(R), three test images must be taken thatinclude two different illumination intensities and two different valuesof t_(l). This leakage current may be calculated either in the capsulecamera or outside the capsule camera. For calculation outside thecapsule camera, the timing parameters for each image are stored in theflash memory or transmitted by wireless. The memory space needed tostore these timing parameters is miniscule relative to the size of theimage. In the above discussion, although an RGB space is used merely forillustrative purposes other color spaces may also be used.

The data for each color component is read out. As a first-orderestimate—i.e., without compensating for thermally-induced leakagecurrent, discussed below—two parameters F_(i) and C_(i) may be stored ortransmitted for each color component R, G or B. Raw color image data mayalso be stored or transmitted to reduce the space requirement ortransmission power. These raw color image data may correspond to a smallnumber of images of red, blue, and green objects illuminated atdifferent intensities. Alternatively, images of a number of objects eachwith a uniform known color may be used to derive these coefficients. Forexample, lighting conditions in which the same RGB ratio is used, butdifferent intensities are provided may be used to solve for the C_(R),C_(G) and C_(B) parameters.

Because both the dark current and the responsivity vary from pixel topixel, the values of these parameters are calculated for each pixel, andcompensation is applied to each pixel according the values of theparameters associated with the pixel. Alternatively, compensation may beprovided for each pixel in a block of pixels, using the same parametervalues (e.g., the same additive and multiplicative factors) calculatedover the pixels of the entire block. A block may be any group of pixels,not necessarily pixels over a rectangular area of the image.

The inventors also recognize that the method of the present invention isapplicable also to cameras used for applications other than as a capsulecamera operating under body cavity conditions. For these otherapplications, the temperature dependences in C_(lR), C_(lG) and C_(lB),C_(R), C_(G) and C_(B), and multiplicative factors F_(R), F_(G), andF_(B) (or their functional forms), may have to be taken intoconsideration. In one embodiment, for example, measurements are taken atmultiple temperatures. At each temperature, the values of F_(k) (or itsfunctional form), C_(lk), and C_(k), k=R, G, or B are solved accordingto the procedure described above.

As discussed above, to make compensation available during operations,the coefficients may be calculated for each pixel at the time ofmanufacture and stored. Alternatively, the images data taken under knownuniform illumination conditions may be stored for subsequentcalculations of the coefficients. As the number of images taken duringoperations in a capsule camera is in the order of tens of thousands, thedata storage requirement for a few or even a few tens of test images forcharacterization is minimal, and may be stored in the on-board archivalmemory (e.g., memory 20). These test data may also be transmitted bywireless to an external recorder, which may be used later in aphysician's examination of the images. Each capsule camera may beprovided an identification number (ID). Based on the ID, the test datamay be provided on and downloaded from a website at the time thephysician examines the captured image data.

The computational requirement (e.g., in terms of circuit area and power)for extracting the coefficients based on the simple relationshipsdiscussed above is relatively low and, either the test image data (e.g.values for each color component) or the coefficients, which may becalculated by on-board circuits, may be easily stored or transmitted. Toillustrate the power and circuit requirements, consider anorder-of-magnitude estimate for a JPEG circuit required to processthirty 300 k resolution images. (There are many selections of JPEGcircuits available from various popular foundries' libraries, eachconsisting of no more than about 50 k gates.) For a 0.09 gm process, theaverage intrinsic gate capacitance is estimated to be:

4 C_(i)WL≅(5 ff/μm²)(0.5 μm)(0.09 μm+0.2 μm)≅3 ff

(The factor of 4 assumes that an equivalent gate has 4 transistors.) The0.2 μm term represents fringing capacitance. For aone-clock-per-code-word JPEG standard widely adopted today, about 30 MHzis needed to run a thirty frames per second real time speed. Therefore,assuming a wiring capacitance of twice the intrinsic capacitance and anactivation rate of ⅓ of all circuits, the power required is estimated tobe:

0.5 CV ²*30 MHz=0.5*(2+1)*3 ff*50 k*(⅓)*(1V)²*30 MHz=2.25 mW

The energy for processing 30 frames is 2.25 mW×1 sec=2.25 mJ, which is aminiscule value within the capability of the batteries of the capsulecamera. By comparison, the equations for calculating the compensationcoefficients above are much simpler than the computational requirementsfor JPEG processing, which includes space-frequency domain discretecosine transform (DCT) operations and entropy encoding, among othersteps specified in ISO 10918-2 JPEG coding standard. Further, the numberof test images required for characterization is normally substantiallyless than 30 images. Therefore, extracting the coefficients in thecapsule camera itself may also be easily achieved in practice.

Thus, the cost of correcting the captured images using the methodsdiscussed herein, and other methods are not cumbersome operations. Inaddition, both the compression ratio and the quality of the compressedimage are more favorable in a corrected image than an uncorrected imagebecause the pixel-to-pixel variations in the uncorrected image arepresented as noise in the image data. Without an elimination procedure,such noise is blended in with the useful information and, consequently,increases the number of the bits in a compressed image. The short rangedifferences (e.g., between adjacent pixels) increase thefrequency-domain data, thus impacting both the compressed file size andthe image quality. During compression, the processing of the noise inthe image data further increases power consumption. For example, in thecase of MPEG processing, a previous reference frame has to bedecompressed in order to code differences between the current frame anda reference frame. The long range differences (i.e., between pixels asubstantial distance away from each other within the sensor array)affect motion estimation in MPEG-like compression. Motion estimation insuch a compression algorithm is affected by both short range differencesand long range differences, because the compression is carried out byspace- and frequency-domain transformations after motion estimation.

A method of the present invention is applicable to characterizing imagesensors outside of the camera. Such a method achieves better quality, abetter compression ratio and provides a lesser power requirement,because the method introduces less noise, allows better matching, andresults in less frequency domain terms. The impact on MPEG compressionis two-folds: both short-range and long-range variations are corrected.The corrections improve compression ratio of short-range, adjacent orproximal pixels within a sensor, with the long-range pixel-to-pixelvariations higher than the short-range variations by an order ofmagnitude. The camera's objective optical elements may also beseparately characterized. Typically, each individual objective opticalelement need not be separately measured. The calibration data for theimage sensor and the objective optical elements may be combined.

In one embodiment of the present invention, an image sensor ischaracterized using one of the methods discussed above (with or withoutproviding a lens and under uniform or known conditions). An image sensortypically has more high frequency components due to compression noise.If a lens is provided, a nearly ideal lens should be used. Otherwise,the lens should be optically characterized offline before use, with itsimperfections taken into account. Correction data is then derived, whichis then associated with the particular sensor. The correction data maythen be provided to the camera manufacturer for subsequent use.

Light from the illumination sources are, to some extent, reflected bythe inner and outer surfaces of capsule housing 10 and other objectswithin capsule housing 10 (e.g., particulate contamination). Suchreflections may be both specular and scattered. Some reflected light maypass into the camera's input pupil to collect on the image sensor. Lightmay reflect multiple times from multiple objects before reaching imagesensor 16.

The spurious signals resulting from reflections of the LED light areindependent of the scene photographed. To characterize the reflectedilluminating light picked up by image sensor 16 from the surface of thetransparent window in capsule housing 10, the capsule camera may beplaced completely in a light-absorbing environment (i.e., no lightreflects from the environment). Under this arrangement, the light sensedat image sensor 16 consists only of the light emitted from the LEDs thatis reflected from capsule housing 10 or from other objects within thecapsule. For simplicity, the following discussion assumes LED lightsources but other light sources could be used instead, such as thosebased on incandescence, fluorescence, field-ionization, phosphorescence,or any other light emission process.

The measured red, green, and blue pixel signals may be expressed as,

$\begin{matrix}{{P_{R} = {{\sum\limits_{i = 1}^{n}{C_{ir}I_{i}t_{LEDi}}} - {C_{lR}t_{l}} - C_{R}}}{P_{G} = {{\sum\limits_{i = 1}^{n}{C_{ig}I_{i}t_{LEDi}}} - {C_{lG}t_{l}} - C_{G}}}{P_{B} = {{\sum\limits_{i = 1}^{n}{C_{ib}I_{i}t_{LEDi}}} - {C_{lB}t_{l}} - C_{B}}}} & (2)\end{matrix}$

where C_(lr), . . . C_(nr), C_(lg) . . . C_(ng), and C_(lb), . . .C_(nb) are parameter relating to the intensity of each LED, I_(l), . . ., I_(n) are the currents in LED_(l) to LED_(n), respectively, and timest_(LEDl), . . . , t_(LEDn) are the durations of the corresponding LEDcurrents. The subtractive terms such as C _(lR) and C_(R) are respectiveleak parameter and offset determined using the previously describedprocedures. Since the LED emitted optical power is roughly linear withcurrent, the reflected light sensed at the image sensor is expected todepend linearly on the currents flowing in individual LEDs integratedover time. For simplicity, equations (2) assume the current I_(i), forLED_(i) is constant but the current terms in equations (2) may also eachbe expressed as an integral of the instantaneous current over time,rather than multiplication of current and time.

Measurements are made with n different combinations of LED currents toallow solving the n independent linear equations for each color.Ideally, each LED is turned on by itself in turn. The n equations forred may be solved using n images, and the values of leakage currentfactor, C_(lR), and offset C_(R) obtained from the image tests ofequations (1). Alternatively, two additional images may be taken and,using equations (2), the leakage current and offset factors may beobtained. In that case, the number of total images needed is n+2.)

The reflections from the outer surface of capsule housing 10 depend onthe medium in which the capsule camera is immersed. Often, in the GItract, the capsule is immersed in a watery liquid. In some cases, thecapsule will be only partially covered in liquid. One set of calibrationdata of LED reflections may be taken with the capsule in a blackcontainer filled with water. Another set of calibration data may betaken without water. When the GI image data are corrected, thecalibration data set which provides optimal correction can be selected.A different set of calibration data may be selected to correct adifferent image region. The suitable correction for an image region maybe indicated by various criteria such as:

-   -   1. Minimize high frequency components of spatial frequency        spectrum.    -   2. Minimize sharp edges in the image, using image processing        algorithms for detecting edges.    -   3. Maximize change between successive images. Usually, the        medium touching the capsule within an image region is the same        from one frame to the next, so that the suitable correction in        each pixel for LED reflections off the capsule housing 10's        outer surface is invariant from one frame to the next. If the        correction is correctly identified (i.e., using the proper        calibration data set), the change in image is maximized.    -   4. Yield only non-negative corrected pixel signals. A negative        value indicates that the wrong calibration data set is applied.

The uniqueness of each LED and the non-uniform light that is cast onobjects by the LEDs—due to both their respective positions and thefinite distances between the objects and the light sources—may also becompensated. Even without knowing the distances inside the smallintestines and inside the large intestine, a correction may still bemade since the light sources' locations and strengths are known and anaverage human being's intestines have certain characteristic range inshape and size.

Tests or characterizations designed to extract the compensationparameters may be carried out during the manufacturing process (e.g.during the assembly process), storing the resulting images and theassociated information or derived coefficients inside the capsule cameraor externally.

Alternatively, the characterization or test may be performed by ahospital technician or by a physician, who runs an automatic testprogram with the capsule camera placed into a container providing thecontrolled characterization environment. The test images with associatedinformation or extracted coefficients may be stored in an on-boardmemory or, if the capsule camera has a wireless transmitter and anantenna, the coefficients database or the images with associatedinformation may be transmitted outside. These data may be retrieved at alater time at a workstation for display and archiving. Thecharacterization data may be archived with a patient's health record orwith the raw (i.e., uncorrected) image data. Alternatively, thecorrected image and the characterization data are stored, with adescription of how the compensation procedures are carried out. Thephysicians may have the freedom to undo the compensation using thecharacterization data.

FIG. 4 shows a method for using a capsule camera using thecharacterization procedures described above, in accordance with oneembodiment of the present invention. As shown in FIG. 4, a capsulecamera including at least a lens system, an image sensor, anillumination system with two LEDs inside the capsule housing, anarchival memory and an output port is prepared (step 401). Next, at step402, the capsule camera is placed in a light absorbing environmentwithout external light sources. Two images are then taken usingilluminations by the illumination system at different relativeilluminations. (The term “relative illumination” refers to the productof exposure time and light intensity.) The images are then stored in thearchival memory. At step 403, the capsule camera is placed in anenvironment of uniform lighting condition using an external lightingsource. Three images of different uniform lighting conditions and pixelprecharge to readout times are captured. The images are then stored inthe archival memory.

Alternatively, at step 402, images are taken each with one LED on byitself in a dark environment. Then, two images are then taken with bothLEDs off for two different time periods. The time-dependent andtime-independent leakage signals as well as the LED reflections arecharacterized. The characterization data is then stored in the archivalmemory. Then, at step 403, one or more images are taken with a uniformwhite background and not LED illumination. (Only one image is requiredto determine responsivity.)

At step 404, the capsule camera is swallowed by a patient to take imagesof the patient's GI tract, using the capsule camera's illuminationsystem as light source. These images are captured into the archivalmemory system. At step 405, after the capsule camera is expelled fromthe patient's body, the capsule camera is recovered, and the capturedimage and the images taken at steps 402 and 403 (if stored) areretrieved through the output port of the capsule camera. The imagestaken at steps 402 and 403 are used to extract the parameter values inaccordance with equations (2) and (1), respectively, forcharacterization or calibration of the capsule camera system. Thecaptured images are then compensated using the extracted parametervalues.

Alternatively, the test or characterization data may be transmittedthrough an output port (e.g., output port 28) or transmitted by wirelessmeans to outside the capsule camera for external storage. FIG. 5illustrates an alternative embodiment of the present invention describedin conjunction with FIG. 4. In the method illustrated in FIG. 5, insteadof an archival memory, the capsule camera is provided with anon-volatile memory buffer and a transmitter. The images taken in thecapsule camera, including the images taken at steps 502 and 503, arefirst stored in non-volatile memory buffer and transmitted to anexternal receiver, where the images are stored for use in the subsequentsteps. In one embodiment, the stored correcting image data or thecoefficients are transmitted to the receiver wirelessly just before thecapsule takes measurements of the patient.

The capsule camera may be marked with an ID and the externally storedtest data or the parameters extracted from the test data, may be madeavailable on websites, by e-mail, in a flash memory, on a VCD or anothermeans. In one embodiment, after the capsule camera is recovered fromtraversing the GI tract, the measurements data may be downloaded to aworkstation, where the test data, or the parameters extracted from thetest data, may be retrieved to compensate the captured image data.Alternatively, an ID or ID information may be embedded in the test orcharacterization image or coefficient data. A technician at aworkstation may use the embedded ID to retrieve the test orcharacterization data or coefficients over the internet, or from anotherstorage media, to perform compensation.

FIG. 6 illustrates an alternative embodiment of the present inventiondescribed in conjunction with FIG. 4. In the method illustrated in FIG.6, instead of an archival memory, the capsule camera is provided withdevice ID and a transmitter and antenna. The images taken in the capsulecamera, including the images taken at steps 602 and 603, are transmittedto an external receiver together with the device ID. The images arestored at an external depository, which may be made available forretrieval over the internet. These images retrieved from the depositoryare used in the subsequent steps.

FIG. 7 illustrates the alignment between an image sensor array (e.g.,sensor array 16) and the footprint of an image produced by an opticalsystem, such as lens 14 of capsule camera 01 of FIG. 1. Ideally, asshown in FIG. 7, if all the components are defect-free and are perfectlyaligned, the optical center of a captured image coincides perfectly withthe center of the sensor array. In practice, however, the center of thecaptured image and the center of the sensor array are not perfectlyaligned. In addition, imperfections typically exist in the components ofthe optical system. Normally, the captured image is designed to beslightly smaller than the size of the sensor array to ensure that theentire image is captured within the sensor array. Using the test imagesfor characterization discussed above, a misalignment of the capturedimage and the center of the sensory array may be derived. For example,under uniform lighting, the average value of a color component in eachcolumn of the sensor array indicates where the edges of the capturedimage are (e.g., scanning from left to right, the column on the right atwhich the intensity transitions from dark to light, and the column onthe left where the image transitions from light to dark). The sameprocedure along the rows indicates where the edges are at the top and atthe bottom of the image.

As another example, when a test image consists of a repetitive pattern,a DCT—such that used in JPEG or MPEG processing—may be used to calculatethe frequency-domain image information. The image footprint is locatedat where the DCT values increase abruptly, corresponding to thepositions at which the pattern becomes clear and discernible. An offsetbetween the optical center and the center of the sensor array (e.g., thedisplacement expressed in x- and y-coordinates) may be stored for laterreference in registers or in the non-volatile memory (e.g., archivalmemory 20). By cropping the operational boundaries of the sensor arrayto the actual image footprint, or a “meaningful” portion of the imagefootprint (i.e., the area where the actual field of view is expected),power is not wasted on the areas in the sensor array outside of theactual useful field of view. In addition, the space required for storingthe image, or the bandwidth required for transmitting the image, is alsoreduced. The workstation utilization rate is enhanced and archivingmanagement is also made easier.

The DCT may be performed in 8×8 blocks for luminance. Therefore, in oneembodiment, to process an area of M columns of pixels,

$\left( {\frac{M}{8} + 1} \right)$

blocks are used for each row. Under such an arrangement, the resolutionis 8 pixels in both row and column directions. Once an 8×8 block isfound where its 64 DCT values start to increase or decrease overprevious blocks, the current 8×8 block is reformed by including the lastcolumn of the previous block in the same row, and a new DCT value iscomputed. If the DCT value increases or decreases further still, theprocedure is repeated to include one further column of that previousblock. This procedure is repeated until the maximum or minimum DCT valueis found, representing where DCT values start to increase or decrease inthe row direction. When applied in the column direction, this methodlocates the pixel at which DCT values start to increase of decrease inthe column direction.

In another example, the edge information may be used to provide thealignment information. Edge enhancement is used in many of today'scommercial cameras or image post processing software. One way to extractan edge is to find the difference in luminance between pixels inadjacent columns. The difference provides edge information in onedirection. The same procedure may be used on corresponding pixels inadjacent rows to get edge information in the direction to the columns.The edges obtained for the column and row directions may be usedtogether.

Illustrated also in FIG. 7 is the fact that, due to individualvariations in the lens, the magnification factor for each lens is notthe same across capsule cameras, thereby resulting in differentfootprints from camera to camera. An alternative representation of theimage footprint may be, for example, two vertices of the rectangularimage footprint. As discussed above, image compensation may be carriedout within the capsule camera (e.g., image processor 18, or bycontrolling the sensor analog gain and providing an offset to each colorcomponents of every pixel.) The storage space required for compensationor characterization parameter values may be provided in a separatenonvolatile flash memory, in a memory provided in anapplication-specific integrated circuit (ASIC) or in the image sensor.Alternatively, the compensation or characterization parameter values mayalso be stored in the same flash non-volatile storage for captured imagedata (e.g., archival memory 20).

FIG. 8A shows a Lambertian curve of a radiation pattern of an LED (apolar plot showing illuminance as a function angle relative to an axisof the LED) and FIG. 8B the luminance as a function of the angle (linearscale). These figures show the imperfections of an actual LED design.

FIG. 9 shows a grid-distortion plot of a lens. As shown in FIG. 9, thedistortion in the middle is minimal (i.e., the actual grid points matchwell with the reference grid) but the distortion becomes pronounced awayfrom the center.

There is a fundamental difference between the design issues posed byFIGS. 8A and 8B and FIG. 9, as large manufacturing variations existamong the LED products. Alternatively, in lens manufacturing, asmeasured by grid distortion, variations are small and deterministic. Theproblem shown in FIG. 9 may be compensated in the most part, even afterdata acquisition by just inverse mapping of the acquired image on thework station or another computer before displaying. The imperfections inFIGS. 8A and 8B are caused by both design imperfections andmanufacturing variations. Such imperfection or variations may beefficiently handled using the methods in FIG. 2-6 above.

In one embodiment, the grid distortion could be compensated using themethods in FIGS. 2-6 by imaging a known pattern, for example a grid, anddetecting an edge (as previously described) to calculate the degree ofdistortion of the image on the sensor surface caused by opticalimperfections (design imperfection as well as manufacturing variations)before displaying. Other problems (e.g., lateral color) may be solvedsimilarly. If deterministic, the image may be compensated on a workstation before the image is displayed, and if the defects are caused byboth by design imperfection and manufacturing variations, only themethods in FIG. 2-6 may be used realistically to compensate to get ahigher fidelity of image arid higher detection rate in case of medicaldevice application.

Accurate reproduction of the captured image is also important.Techniques similar to those described above may be applied to alldisplay devices, including television sets. In recent years, the displayindustry has increased image resolution from standard definition (SD) tohigh definition (HD) and ultra-high definition (UHD), and other imageresolutions. In addition to increases in resolution, display devicesalso have their color resolution improved from 8 bits to 9 bits, andeven 10 bits, per color component. Thus, the need to controlmanufacturing variations has increased significantly and, consequently,the cost for ensure display quality has also significantly increased.

According to one aspect of the present invention, to provide acalibration of a display device, standardized color patterns designedfor the entire range of displayable colors are provided as images fordisplay on the display device under controlled operating conditions.This step may take place as a manufacturing step, or repeated at asubsequent time when necessary. In many display devices, their accuracyin color/intensity reproduction varies with the operating temperature.Where the color/intensity reproduction accuracy varies with theoperating temperature, the calibration may take into account theoperating temperature as an operating condition. To take into account ofthe operating temperature, the display device preferably includes athermometer configured to measure the operating temperature of thedisplay panel.

During calibration, a diagnostic system detects the intensity of eachcolor component in each pixel of the image displayed on the displaydevice. The diagnostic system includes sensors (e.g., photodiodes,filters) for determining the colors and the intensities of each pixelover a range of operating ranges. At each calibration operating point,the intensity of each color component in each pixel is compared againstits corresponding expected value to determine a difference, if any. Thediagnostic system may then provide corrective data based on the measureddifferences. The corrective data may be, for example, in the form of acompensation value to the input value of each color component necessaryfor achieving the desired color and intensity. The corrective data isthen recorded in a nonvolatile memory, so as to allow the necessarycorrection to each pixel to be made in future display operations.

According to one embodiment of the present invention, the display devicemay be provided means for measuring temperature (“thermometer”, such asa thermal couple). The operating temperature may be considered anoperating parameter. Typically, the operating temperature in a displaydevice is higher than the ambient, depending on the display device'spower consumption, thermal resistance and ambient temperature and airflow. Compensation that takes into account of the operating temperaturemay be desirable in some display device.

According to one embodiment of the present invention, the correctivedata may be stored in non-volatile memory in any suitable format. Forexample, the necessary compensation value for each color component ofeach pixel over entire ranges of input values and operating temperaturesmay be provided in a table form. Prior to displaying an image, the tableis accessed for each color component of each pixel, using the inputvalue and the measured operating temperature. The compensation valuethus obtained—after interpolation or extrapolation, if necessary—isadded to corresponding input value to prepare the image to be displayed.Alternatively, according to one embodiment of the present invention,after curve-fitting the corrective data to one or more correctivemodels, such as a linear or polynomial functions of the input value andtemperature, the model parameters (e.g., offsets and coefficients of apolynomial function) are then stored in the non-volatile memory. In sucha case, prior to displaying the image, the display device computes acompensation value for each color component of each pixel based on thecorresponding input value and the operating temperature. The image isthen displayed after adjusting each input value based on the computedcompensation value.

Many variations of the above approaches are available. For example, atrade-off that reduces the memory requirement for storing the correctivedata is available by storing the corrective data, rather than on aper-pixel basis, on the basis of groups of neighboring pixels. Accordingto that format, using the group-specific corrective data, compensationis provided for each pixel. For example, each pixel may be compensatedby an amount that is derived from the stored corrective data based on aweighted average of the input values. If corrective data is stored on agroup-specific basis, in some display devices, it may be advantageous totake into account high contrast condition with the group of pixels(e.g., an edge with a transition of brightness) in determiningcompensation for each pixel. In another variation, the input value ofeach color component and temperature is used to look-up a correctedvalue. The compensation is applied by displaying the image using thecorrected values thus obtained, replacing the input values.

Another approach would be a hybrid form of table look-up andinterpolation or extrapolation techniques. For example, the correctivedata may be stored in the form of a table of corrected values forselected input ranges and selected temperature ranges, together withvalues of the model parameters. In such a case, the actual correctedvalue for each color component can be computed from interpolation orextrapolation from the values obtained from the table using the modelparameter values.

Compensation can be carried out in the digital or the analog domain. Forexample, FIG. 10 shows the structures in display device 1100 providedfor compensating pixel values in the digital domain, according to oneembodiment of the present invention. As shown in FIG. 10, input valuesfor an image to be displayed is received into image processing unit1101, which retrieves the corrective data or parameters fromnon-volatile memory 1102. Image processing unit 1101 then computes—ifnecessary—and applies compensation necessary to each color component ofeach pixel of the image. The resulting corrected data is then providedto display driver 1103, which provides the control and data signals ofdisplay hardware 1104.

Alternatively, FIG. 11 shows the structures in display device 1200provided for compensating pixel values in the analog domain, accordingto one embodiment of the present invention. As shown in FIG. 11, afterconventional image processing, input values for an image to be displayedare provided to display driver unit 1203—which may contain both digitaland analog circuits—and non-volatile memory 1202. The input values areused to retrieve corrective data from non-volatile memory 1202. Displaydriver unit 1203 then applies compensation (e.g., adjusts the gain andoffset at each pixel) necessary to accurately reproduce each colorcomponent of each pixel of the image to the desired value. Based on thecorrected values, the display driver unit 1203 provides control and datasignals to display hardware 1204.

According to one embodiment of the present invention, rather thanproviding the compensation in the display device itself, another device(e.g., a computer or a set-top box providing the image data to thedisplay device) may perform the compensation on the image data beforeproviding the compensated image data to the display device. In thatembodiment, the device performing the compensation retrieves thecorrective data from a non-volatile memory of the display device.

The above detailed description is provided to illustrate the specificembodiments of the present invention and is not intended to be limiting.Numerous variations and modifications within the scope of the presentinvention are possible. The present invention is set forth in theaccompanying claims.

We claim:
 1. A display system which receives input image data,comprising: a non-volatile memory containing corrective data forcompensating the input image data; display hardware for displaying animage; and an image processing circuit which (a) based on the inputimage data, retrieves the corrective data from the non-volatile memory;and (b) applies the retrieved corrective data to each pixel in the inputimage data to generate data signals for displaying the image in thedisplay hardware.
 2. The display system of claim 1, wherein thecorrective data in the non-volatile memory is generated from acalibration step during manufacturing by a diagnostic system detectingan intensity of each color component in each pixel of the displayhardware.
 3. The display system of claim 2, wherein the corrective datastored in the non-volatile memory enables compensation to the inputimage data over a range of intensities for each color component of eachpixel displayed by the display hardware.
 4. The display system of claim1, wherein the display device further comprises a thermometer formeasuring an operating temperature in the display hardware, and whereinthe corrective data enables compensation of the input image data basedin part on the measured temperature.
 5. The display system of claim 1,wherein the corrective data is stored in the non-volatile memory in atable format.
 6. The display system of claim 5, wherein the correctivedata corresponds to the corrected values for replacing correspondingvalues in the input image data.
 7. The display system of claim 5,wherein the corrective data corresponds to compensation values to beapplied to corresponding values in the input image data.
 8. The displaysystem of claim 5, wherein the corrective data corresponds to correctivevalues for predetermined values of input image data, and whereincorrective values for the received input image data are interpolated orextrapolated from the corrective values for the predetermined values. 9.The display system of claim 5, wherein the corrective data correspondsto corrective values for one or more predetermined values of input imagedata and values of parameters of a model, and wherein corrective valuesfor the received input image data are generated from the values of theparameters based on the corrective values for the predetermined values.10. The display system of claim 1, wherein the corrective datacorresponds to values of parameters of a model from which correctivevalues are generated based on the corresponding received input imagedata.
 11. The display system of claim 10, wherein the model comprises alinear function or polynomial function.
 12. The display system of claim1, wherein the corrective data comprises corrective values eachapplicable to one of a plurality of specific groups of neighboringpixels in the image hardware.
 13. The display system of claim 1, whereinthe corrective data is applied to the display hardware in the digitaldomain.
 14. The display system of claim 1, wherein the corrective datais applied to the display hardware in the analog domain.
 15. A methodfor correcting input image data received into a display device formanufacturing variations, comprising: generating corrective data from acalibration step during manufacturing by a diagnostic system detectingan intensity of each color component in each pixel of display hardwarein the display device; storing the corrective data into a non-volatilememory in the display device; at a subsequent time, (i) retrieving thecorrective data from the non-volatile memory; (ii) applying thecorrective data to each pixel in the received input image data; and(iii) generating data signals for the display hardware, based on thecorrected input image data.
 16. The method of claim 15, wherein thecorrective data stored in the non-volatile memory enables compensationto the input image data over a range of intensities for each colorcomponent of each pixel displayed by the display hardware.
 17. Themethod of claim 15, wherein the display comprises a thermometer formeasuring an operating temperature in the display hardware, and whereinthe corrective data enables compensation of the input image data basedin part on the measured temperature.
 18. The method of claim 15, whereinthe corrective data is stored in the non-volatile memory in a tableformat.
 19. The method of claim 18, wherein the corrective datacorresponds to the corrected values for replacing corresponding valuesin the input image data.
 20. The method of claim 18, wherein thecorrective data corresponds to compensation values to be applied tocorresponding values in the input image data.
 21. The method of claim18, wherein the corrective data corresponds to corrective values forpredetermined values of input image data, and wherein corrective valuesfor the received input image data are interpolated or extrapolated fromthe corrective values for the predetermined values.
 22. The method ofclaim 18, wherein the corrective data corresponds to corrective valuesfor one or more predetermined values of input image data and values ofparameters of a model, and wherein corrective values for the receivedinput image data are generated from the values of the parameters basedon the corrective values for the predetermined values.
 23. The method ofclaim 15, wherein the corrective data corresponds to values ofparameters of a model from which corrective values are generated basedon the corresponding received input image data.
 24. The method of claim23, wherein the model comprises a linear function or polynomialfunction.
 25. The method of claim 15, wherein the corrective datacomprises corrective values each applicable to one of a plurality ofspecific groups of neighboring pixels in the image hardware, and whereineach corrective value is applied to each pixel of the correspondinggroup of neighboring pixels.
 26. The method of claim 15, wherein thecorrective data is applied to the display hardware in the digitaldomain.
 27. The method of claim 15, wherein the corrective data isapplied to the display hardware in the analog domain.